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pre-Gaussian conditions 

Shahar MendelsorPIS and Joel ZinrM] 

ANU & Technion I.I.T and Texas A&M University 

Abstract: We show that a modified Empirical process converges to the lim- 
iting Gaussian process whenever the limit is continuous. The modification de- 
pends on the properties of the limit via Talagrand's characterization of the 
continuity of Gaussian processes. 



1. Introduction 

Given a class of functions F C £2(^)1 a now standard method to use (iid) data 
to uniformly estimate or predict the mean of one of the functions in the class, is 
through the use of empirical processes. One has to bound the random variable 
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One possibility that comes to mind is to use the fact that there is a bounded Gaus- 
sian process indexed by F to bound y/n\\P n — P\\p. To illustrate the difficulty one 
encounters, note that to use the classical Central Limit Theorem in finite dimen- 
sions, one only needs finite variance or, equivalently, the existence of the (Gaussian) 
limit. However, when working in the infinite dimensional situation there are some- 
times non-trivial side conditions other than just the set being pregaussian. Those 
are connected to the random geometry of the set F that are needed to ensure the 
existence of a useful bound. For example, one such situation is when the class is 
a collection of indicators of sets. If such a class does not satisfy the VC (Vapnik- 
Cervonenkis) condition, then, in addition to knowing that the limiting Gaussian 
process is continuous, one has to check, for example, the following: 

log^cn^, x n} :CeC} ^ . n Quter probabmty 

In this note we try to get around this problem by looking at a variant of the 
standard empirical process for which only the existence of the limiting Gaussian 
process is required to obtain both tail estimates and the Central Limit Theorem 
for the modified process. 

The motivation for our study were the articles 00, which focus on the following 
problem. Consider a density p(x) on R which has a support that is a finite union 
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of intervals, and on this support p satisfies that c™ 1 < p(x) < c and \p(x) — p(y)\ < 
c\x — y\. It was shown in 0, Q that under this assumption there is a histogram 
rule P n for which the following holds. If F is a P-pregaussian class of indicator 
functions, or if F is a P-pregaussian class of functions bounded by 1 which satisfy a 
certain metric entropy condition, then \fn(P n — P) converges weakly to the limiting 
Gaussian process. 

It seems that the method used in 0, @| can not be extended to densities in 
M 2 , and even in the one-dimensional case the convergence result holds for a very 
restricted set of densities. Thus, our aim is to find some other empirical estimator 
for which the existence of the limiting Gaussian would imply convergence as above. 
Our estimator is based on Theorem 1 in 0] (see also Theorem 1.3 j^j and Theorem 
14.8 i). 

We begin with several facts and notation we will use throughout this note. If 
G is a (centered) Gaussian process indexed by the set T, then for every s, t 6 T, 

P2 (s,t) = (E(G s -G t ) 2 ) 1/2 . 

To prove that a process (and in particular, the modified empirical process we 
define) converges to the Gaussian process indexed by P, we require the notion of 
asymptotic equicontinuity. 

Definition 1.1. A net X a : il a — > £oo(T) is asymptotically uniformly p-equi- 
continuous in probability, if for every e, r\ > 0, there exists a S > such that 

limsup a Pr*( sup \X a (s) - X a (t)\ > e) < rj. 

p(s,t)<S 

Theorem 1.2 ([HI, 1.5.10). Let G be a Gaussian process and let X a be a net of 

random variables with values in ^ (T) . Then there exists a version of G which is a 
tight Borel measurable map into £oo(T), and X a converges weakly to G if and only 
if 

(i) the finite dimensional distributions of X a converge weakly to the corresponding 
finite dimensional distributions of G, 

(ii) X a is asymptotically uniformly equicontinuous in probability with respect to 

92, 

(iii) (T, P2) is totally bounded. 

The main technical tool we require is generic chaining which was developed by 
Talagrand (see [ll| for the most recent survey on this topic). 

Definition 1.3. For a metric space (T, d), an admissible sequence of T is a collection 
of subsets of T, {T s :s> 0}, such that for every s > 1, \T S \ = 2 r and \T \ = 1. For 
P > 1, we define the 7 P functional by 

00 

7p (T, d) = inf sup V 2 s /?d(t, T s ), 

where the infimum is taken with respect to all admissible sequences of T. 

For every integer s define the function 7r s : T — > T s , which maps every t £ T to 
a nearest element to t in T s . 

Using the 7 P functionals one can bound the supremum of a process which satisfies 
an increment condition. As an example, recall the well known Bernstein inequality. 
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Theorem 1.4. There exists an absolute constant c for which the following holds. 
Let (Q, S, P) be a probability space, let f € L^^P) and let X%, . . . , X n be indepen- 
dent random variables distributed according to P. Then 



Pr 



> t > < 2exp -cnmin 



t 2 t 



/IIL'll/lk 



When one combines Bernstein's inequality with the generic chaining method, 
the following corollary is evident: 

Corollary 1.5 (|ll|). There exists an absolute constant c for which the following 
holds. Let F be a class of functions on a probability space (Q, P). Then, for every 
integer n, 

FIIP Pll <r f l^ElllMl i 71(^,11-1100 )' 

E \\P n - P\\ F < c -= 1 

\ y/n n 

A key result, which follows from the generic chaining method, is that the expec- 
tation of the supremum of the Gaussian process indexed by F and has a covariancc 
structure endowed by L%{P) is finite if and only if 72 (F, L2(P)) is finite. Moreover, 
the result is quantitative in nature. 

Theorem 1.6. There exist absolute constants c\ and C2 for which the following 
holds. Let F be a class of functions on (Q,P), and let G be the Gaussian process 
indexed by F. Then, 

Cl j 2 (F,L 2 (P)) <E sup \G f \ <c 2l2 {F,L 2 {P)). 

The upper bound is due to Fernique while the lower bound in due to Ta- 
lagrand [l(| • A proof of both parts can be found in . From here on we denote 
Esu P/ \G f \ by E\\G\\ F . 

In a similar fashion one can formulate a continuity condition for the Gaussian 
process indexed by F using the generic chaining machinery. 

Theorem 1.7 ([HI Theorem 1.4.1). Consider a Gaussian process, {Gf : f G F}, 
where F is countable. Then the following are equivalent: 

(1) The map f — > Gf(uj) is uniformly continuous on (F,\\ ■ \\ 2 ) with probability 
one. 

(2) We have 

limE sup \G f -G f >\ = 0. 

||G / -G! / /|| 3 <6 

(3) There exist an admissible sequence of partitions of F such that 

lim sup V 2 s ' 2 d{f,F s ) = 0. 

J s>s 

Note that the admissible sequence in (3) can be taken as an almost optimal ad- 
missible sequence in the definition of 72 (F, || • H2), at a price of an absolute constant. 
Indeed, by combining the two admissible sequences (T s ) and (T' s ) - the first - an 
almost optimal one from the definition of 72 and the second from (|1.7p , we obtain a 
new admissible sequence for which T s U T' s C T" +1 . Thus, we may assume that the 
almost optimal sequence in (3) satisfies that supj gF ^^l 2 s / 2 d\\.\\ 2 (t, T s ) ~ E||G||_f. 

Finally, a notational convention. Throughout, all absolute constants will be de- 
noted by c, C or K. Their values may change from line to line. We write a ~ b if 
there are absolute constants c and C such that ca < b < Ca. 
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2. The main theorems 



Let F C L 2 (P) and assume that there exists a Gaussian process indexed by F 
such that E||G|| F < oo. By TheoremHH j 2 (F,L 2 (P)) ~ E||G|| F , and let {F s )f =0 , 
F s C F be an almost optimal admissible sequence with respect to the L 2 {P) norm 
as described above, that is, 



and 



sup^2 s / 2 ||/- 7 r s (/)|| L2(P) <c 72 (F,L 2 (P)), 



]™L SU P E 28/2 11/ - ^(/)Hi2(P) - 0. 

g>S 



Set A s (/) = 7r s (/) — tt s _i(/). For every s, / and A > consider a truncated part 
of A s (f), defined by A' s (/, A) = A s (/)1{|a s (/)|<a}- As will be made clear shortly, 
the truncation level A depends both on the specific / <G F, as well as on the size of 
the sample n and the index s. 

Lemma 2.1. There exist absolute constants c\, c 2 and C3 for which the follow- 
ing holds. Let F be a class of functions on the probability space (f2, P), and set 
X\,...,X n to be independent, distributed according to P. Let (F s ) r ^L be an ad- 
missible sequence in F with respect to L 2 (P), and for every f £ F set A — 
CiV n ~\\A s (f)\\ 2 /2 s / 2 . Then, for every u > 1/2 and every integer s, with probability 
at least 1 — 2exp(— C22 s min{u 2 , u}), for every f 6 F. 



(2.1) 



1 

-V(A' s (/,A))(A-)-EA' s (/,A) 



< C3U 



2 5/2 ||A,(/)ll 



Proof. Let c\ and c 2 be constants to be named later. By Bernstein's inequality Q, 
for t — c\2 s / 2 u\\ A s (f)\\ 2 /^/n, it is evident that for any A > 0, 



Pr 



1 ™ 

-^(AU/,A))(X i )-EA' s (/,A) 



> t 



<2exp —cn min 



|A' S (/,A)|| 



Since ||A^(/,A)|| 2 < ||A S (/)|| 2 and A = c 2y fn\\ A s (/) 1 1 2 /2 s / 2 , then for the choice of 
t, it follows that with probability at least 1 — 2 exp(— 032 s min{w 2 , u}), 



(A' s (/,A))(X i )-EA' s (/,A) 



;.=i 



< C4M 



2 S/2 ||A S (/)|| 



where C3 and C4 depend only on c\ and c 2 . Thus, for an appropriate choice of these 
constants and since |{A S (/) : / S F}\ < \F S \ ■ \F a -i\ < 2 2 the claim follows. □ 

Note that there is nothing magical with the lower bound of 1/2 on u. Any other 
absolute constant would do, and would lead to changed absolute constants ci, c 2 
and C3. 

Using the Lemma we can define a process $ n for which \\P n {^ n ) — P\\ F < 
c¥*\\G\\f / \/n, an( i thus, the fact that the limiting Gaussian process exists suffices 
to yield a useful bound on the way in which the empirical estimator approximates 
the mean. 
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Definition 2.2. Let A(/,n,s) = A = c ov ^|| A S (/)|| 2 /2 S / 2 , where Co was determined 
in Lemma |2.1[ and for each sq let 

oo 

*«,«,(/) = E A ^' A ) 



and 



$„(/) = E A ',(/' A ) 



Theorem 2.3. There exist absolute constants C\ and c 2 for which the following 
holds. Let F and X±, . . . , X n be as above. Then, the mapping $„ : F — ► Li(P), and 
for every u > 1/2, luii/i probability at least 



1 — 2 exp(— ci min{u 2 , u}), 



for every f G F 



(2.2) 



1 - 

- E (*»(/)) (**)" E/ 



< c 2 (u + 1) 



E||G||j 



and afco mf/i probability at least 1 — 2exp(— c 1 2 Sn min{it 2 ,u}) 



(2.3) sup 



-E(^o(/)(^)-E*n >So (/) 



<^sup E 2 S / 2 ||A S (/)|| 2 . 



Proof. Without loss of generality, assume that € F and that 7To = 0. Let (F s ) ( ^ 1 
be an almost optimal admissible sequence, and in particular, by Theorem 11.61 



su P E2 s/2 ||A s (/)|| 2 < KE\\G\\ F 

for a suitable absolute constant K. 

Note that as 7Tq(/) = for every f £ F one can write / = ^^ =1 A S (/). 
Let us show that $„, and therefore <5„,s > are wen defined and maps F into 
Li(P). Indeed, since Yl^Lx A a (/) converges in L^P), it suffices to prove that 
E^i(A s (/)- K(f, A)) = ET=i A "C0 converges in L±(P). Observe that for every 



(2.4) 



E|A' s '(/,A)|=E|A s (/)|l { | As(/) | >A} 

l|A s (/)|| 2 (f 
2 S/2 ||A S (/)|| 2 



< ||A S (/)|| 2 (Pr (|A S (/)| > A)) 1 / 2 < HM^i 



< 



Since X^^Li 2 S ^ 2 || A s (/)|| 2 converges for every /, it implies that <i> n is well defined 
and takes values in L\. 

By Lemma \2. 11 with probability at least 



(2.5) 1-2 E exp(-ci2 s min{w 2 ,-u}) > 1 - 2exp(-c 2 2 s ° min{u 2 , it}), 



s=s + l 
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su P 

^ F s=s + 1 



n 

-^(A' S (/,A))(X,)-EA^(/,A) 

71 ^— ' 



OO 



<^fsup £ 2 S / 2 ||A S (/)|| 

^/^ s =r+i 



and when s = we'll use that by Theorem 11.61 this last quantity is 

E||G|| F 

< C4W = — . 



Hence, with that probability, for every / 6 F, 
- £(*„(/)(*) -E/) ^ -J2(^(f)(Xi)-E^ n (f)) 

OO 



. E||g||F 

< C5M = h 



s=l 



E/-E4 n (/)l 
E||G|| F 



where the last term is estimated using the same argument as in (|2.4[) and the 
inequality f2~2]) in Theorem 1231 

We also have that with the promised lower bound on the probability, 



sup 



1 n 

-E( $ ^o(/)(^)-E$n, So (/) 



<^su P E 2*/ 2 ||A s ( / )|| 2 . 



Next, we prove a limit theorem for {-\/re(-fn — P)(^n)(f) ■ f S F} and show 
that we can replace E$„(/) with E/ and still obtain a limit theorem. For this we 
need to prove an inequality for the oscillation of the process y/n(P n — P) ($„(/)). 
To that end, define Q n := V" (Pn ~ -P)- 

Proposition 2.4. Let F be a class of functions on (Q,P), such that the Gaussian 
process indexed by F exists and is continuous. is as above, then for any rj > 0, 



lim lim Pr sup Q„(*„(/) - *»(/)) > 7? = 0. 
\||/-/|| a <i 



Proof. By the definition of <£>„ which uses an almost optimal admissible sequence, 
for every S > there is some so such that 



supE 2s/2 ||/-7r s (/)l|2<<5, 



hence for any /,/ 6 F, ||vr So (/) - 7r So (/)|| 2 < 25 + \\f - f\\ 2 - Using the notation of 
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TheoremES put $„, So (/) := E^ So +i K(f, A) and # n , So = $„-$„, So . Therefore, 



On $„(/)-$„(/) 



I : = Pr sup 

\II/-/||2<<5 

= Pr sup 

\ Il/-/ll2«5 



+ Qn(*n,a q (/))-Qn(*n, ao (/)) 



> 77 



QT.(*n,-o(/)-*n,»o(/) 



> 77 



< Pr sup 

\l! 7r s /~ 7T ' s o/ll 2<3 ' 5 



Qn(^(/)-*^(/) 



> 



+ 2Pr fsup|Q„($„ >S0 (/))| > |J 
:= (77) + (777) 

From the proof of Theorem 12.31 by integrating tail probabilities 

E sup |Q„ ($„..„(/)) I <csup V 2 s / 2 ||/-7r s (/)|| 2 

which by Theorem I1.7f 3) and our choice of the admissible sequence converges 
to as so ~~ * 00 ■ Furthermore, by the finite dimensional Central Limit Theorem 
lim5_»o lim ?woo (77) = 0, which completes the proof. □ 

Hence, we know that Q n is asymptotically uniformly equicontinuous. We'll now 
prove the other necessary ingredients needed to show that Q n converges to the 
original Gaussian process. 

Proposition 2.5. Let F and $„ be as in Proposition\2.4\ Then the following holds: 



(i) lim^oo V^sup /eF |E$„(/) - E/| = 0, 

(ii) For every f £ F, lim n ^oo ||$»(/) - /|| 2 = 0, 

1$ ( f)(X )\ 2 

(iii) For every f G F, lim^-oo E max 7 <„ 1 " UA jn = 0. 

n 

Proof. 

1. Let s be an integer to be named later and set 

A = Co V^||A s (/)|| 2 /2 s / 2 

as in Lemma \'2. II In particular, the set {A s (/) : s < so, / G F} is finite and 
for every / e F, 



V^E|A S (/)|1 { |A.(/)|>A} 



< 



2 s/2 



co||A s (/)|| 



-E|A s (/)| 2 l { | As(/) | >A} 



which, by the definition of A tends to as n tends to infinity. Hence, for every 
fixed so, 

lim f^E|A s (/)|l {|As(/)|>A} =0. 
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Therefore, for every sq, 
lim Vnsup|E$ n (/)-E/| 

n—Kx j e p 

/ s N 

< lim ^sup ^E|A S (/)|1 { | A3(/) | >A} + e I a S (/)|1{|a s (/)|>a } 



< 



c 2 sup£2 s / 2 ||A s (/)|| 2 , 



^ F s>s 



where the last inequality is evident from (|2.4j) and the choice of a suitable 
absolute constant c 2 . 
2. Again, we shall use the fact that for every fixed / and s, A depends on n and 
tends to as n tends to infinity. Clearly, for every fixed So, 

11/ - *n(/)||2 < £ l|A.(/)l { | A .(/)|>A}||2 + E W A s(f)h 

s<s s>s 

< X: ||A s (/)l { | As(/)>X} || 2 + C3 7 2 (F,L 2 ) £ 2-*/ 2 

s<so s>so 

For an absolute constant C3. Indeed, this follows from the fact that for every 
s, 

OO 

2 S/2 ||A.,(/)|| 2 < ^2 S / 2 ||A S (/)|| 2 < c 3l2 (F,L 2 ), 

s=0 

and of course the constant C3 does not depend on s. 
Hence, for every fixed / G F, 



lim sup || /-$„(/) || 2 < c 3l2 (F,L 2 ) Yl 2 



-s/2 

s>s 



for every sq, and this last quantity goes to zero as sq — ► 00. 
3. If /(X) is square integrable then for any b > 0, 



limsupEmaxffi^ < lim sup e£ + i £ / a (X,)l { |j , x , 



= Ef 2 (X)l {mx)>b} . 

Since the left hand side does not depend on b and the right hand side converges 

1 f ( x ■ ) 1 2 

to zero as b tends to 00, linin^oo E maxj<„ 1 - n jn — 0. Therefore, to complete 
the proof it suffices to show that 

l/^O-ou/Xx,)! 2 n 



E max ■ 

j<n n 

But, using (2), 

j<n n n 

= E|(/-$„(/))(X)| 2 ^0. n 
The final ingredient we require is the following result on triangular arrays. 
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dent random variables for which linin^oo Emaxj<n = 0. Then, for every 8 > 0, 



Lemma 2.6. For each n, let {Cn,j}^—i by nonnegative, square integrable, indepen- 

Um™E"=iE^, i %„,,> 5} = 0. 
Proof. Consider the stopping times 



T = T„ = 



infjfc < n : > (5} if max r <„ ( n r > 5 
oo if max r <„ r < 5. 



Then, (see [lH) for every n 

n 

„<oo} — 

5_n 2=1 
n 

n 

> J2 E £,lHin <>s} Pr(max^,i < S) 



The result now follows, since the hypothesis implies that this last probability con- 
verges to one as n tends to infinity. □ 

We now can conclude 

Theorem 2.7. If the Gaussian process indexed by F is continuous then 
{V^(Pn(Mf)) -Pf):f€F} converges to {Gf : / e F}. 

Proof. By Theorem 11.21 and Proposition 12.41 we only need to show that 

(i) the finite dimensional distributions of Q n converge to those of G and 

(ii) {F,p2) is totally bounded. 

For (i) we need to check that for any C F, 

(Q»(3n(/l)),...,Qn (*»(/*))) 

converges in distribution to (G f x , . . . ,Gf k ). To see this we apply the Cramer- Wold 
device, that is, by noting that to show the convergence in distribution, we only have 
to check that the characteristic function (on M. k ) converges, and hence it suffices to 
show that any finite linear combination of {Q n ($n(fi))}i=i> say, a iQn{®n(fi j) 

converges in distribution to 2J i=1 a iGh - To verify this, recall the classical Central 
Limit Theorem for triangular arrays (see, e.g, [5J or [1| Theorem 3.5). Namely, it 
suffices to prove that 

(a) for any n > 0, lim^oo Pr(maxj<„ | a i®n(fi(Xj))\ > n) — and 

(b) lim^oo Var((X)* =1 Oi$ n (/<))i { | ^ (A)l»7p = Var S*=i 

(a) follows from Proposition ^ . 5f iii) and (ii) and (b) follows from l2.5f n) and Lemma 



(ii) follows from the assumed continuity of {Gf : / <E F} with respect to P2 (see p. 
41 [lj). □ 
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3. Changing the level of truncation 

The question we wish to tackle here is whether it is possible to find different "uni- 
versal" truncation levels instead of y/n, and still have a process \& which is tight, 
and satisfies that n~ x X^Li W'nC/)) P^) uniformly approximates Ef (that is, one 
can replace E$„(/) with Ef). We show that such a uniform level of truncation has 
to be asymptotically larger than ypn. 

Definition 3.1. Given a class of functions F and a non-decreasing sequence of 
positive numbers, b = {6„}^L 1 , let 

oo 

$n,b = ^2 A s(f)t{\A 3 (f)\<b n \\A 3 (f)\\ 2 /2^}- 
s=l 

Definition 3.2. A sequence of processes {U n (f) : f G F} is said to be stochasti- 
cally bounded if for every e > there is a constant C < oo such that 

Pr(sup|[/„(/)|>C)<e. 

Theorem 3.3. Assume that {b n } n is an increasing sequence of positive numbers 
and that the probability space, (0,<S,P), is continuous. Assume also that for every 
pregaussian class of functions on (Q,S,P), the process 

{sup V / ^|P„(*n, & (/))-E/|} Il 

feF 

is stochastically bounded. Then, there exists S > such that inf„ -^L > S. 

Proof. Clearly, if {y/n (P^($ ni f,(/)) — Ef) : f <G F} is based on an independent 
copy, {X'j}, then {sup^- 6J? y/n\P^{^ n ^{f)) — Ef\}^ =1 is also stochastically bounded. 
Hence, the difference is stochastically bounded, and thus, 

{sup V^\Pn($n,b(f)) - E$ n , 6 (/)|}„ 

is stochastically bounded, implying that -^/nsupyg^ |E/ — E<&„ ; (,(/)| is bounded. 

In particular, for every nonnegative / € L 2 {P), if we let F — {/, 0}, then the 
sequence {y/n\Ef— E$„ i b(/)|}^ =1 is bounded. Note that in this case we may assume 
that ir s (f) = f for s > 1 and tt (/) = 0, implying that v / ^ E / 1 {/>6 n ||/|| 2 /V2} is 
bounded. 

Observe that this implies that y/nEfl{f >bn y is bounded. Indeed, choose bk 
such that ||/l{/>b fco }||2 < V%- Applying the above to the function h = fty >bka j, 
it follows that \\h\\2 < V% and 

VnEhl {h>bn} = v^E/l {/>6fco} l {/1{/>6fco}>6n} = v / HE/l {/>6max(fcon)} . 

Hence, y/nEfl^f >bn } is bounded, as claimed. 

For every sequence {at}k for which ^ fc |a.fc|/fc < oo, consider a function / with 
Pr(/ = bk) = biiffi. Such a function exists by the continuity of the probability 
space (fi,«S, P). Then, 
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Therefore, E/l{y >bfc } = Yl,i>k htf. zp> implying that for every sequence {a,k}^ =1 as 

above, sup fc Vk^2 l>k < oo. 

Consider the Banach spaces B\ and Bi, endowed with the norms ||{dfc}||i 

: = J2T=i TT and Il{ a *;}ll2 : = su Pfc>i ^Yji>k 757- Note tnat tne identity map 
I : B\ — > i?2 is bounded using the Closed Graph Theorem. Indeed, for A n :— 
{a n>k }f =1 ,B := {b k }^ =1 and C := {c k }f =1 assume that \\A n - and 

||A„ — C| 1 2 — * 0. These conditions respectively imply convergence coordinate- wise, 
that is, for every r, lim n ^oo a n ,r = b r and limn— «x> dn,r = c r . Thus, B = C, and 
the graph is closed, implying that the map is bounded. 
Therefore, there exists a constant, C, such that 



(3.1) -p^^<^W 

Applying (|3.ip to the sequence for which the n th term is one and others zero shows 
that for n > 1: 

no n n 

from which the claim follows. □ 
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